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Method and system for navigating in real time in three- 
dimensional medical image model 

Field 

The invention relates to a system and a method for navigating in 
5 real time in a medical image model within a three-dimensional virtual work- 
space. 

Background 

Medical diagnosis and surgical planning typically comprise studying 
two-dimensional Images of a patient on an illuminated light box or a computer 

10 displayrfo7 example. The two-dimensional images are, for example, MRI 
(magnetic resonance imaging)- slices of a target area of the patient. MRI is 
used to visualize some procedures such as brain surgery. In order to make the 
diagnosis or planning treatments the two-dimensional image slices are rou- 
tinely studied. However, the understanding of the target area of the patient 

15 based on the two-dimensional image slices Is time-consuming and a difficult 
process. One reason for that is that the visualization Is a two-dimensional 
process while the actual surgical procedure is three-dimensional. 

Minimally invasive treatment of the human body is becoming popu- 
lar. The treatment can be planned by virtual reality visualization of the treat- 

20 ment area. Known minimally invasive surgical procedures are often visually 
guided, but such methods often do not permit visualization within the target 
tissue or organ. Intuitive real-time three-dimensional visualization of the tissues 
would provide accurete guidance of therapy. 

Systems for representing infomriation as rendered three-dimensional 

25 images have proved to be suited to representing large amounts of infomriation 
and / or complex information in an efficient and in a compact manner. Users 
can often more easily understand the displays produced within such infonna- 
tion visualization systems than other conventional representations. The user 
viewpoint within the visualization systems is the viewpoint to which the three- 

30 dimensional representation of the three-dimensional Image model Is rendered. 
The rendering system responds to input from the user to change the desired 
viewpoint accordingly. When a new viewpoint position and / or distance are 
input by the user, the rendering system re-renders the view appropriately. 
When real time or near-real time rendering is provided, the user is able to up- 

35 date the viewpoint and study the result immediately. 
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However, the known visualization systems for medical image mod- 
els are uncomfortable for the users. The user is npt able to navigate through 
the three-dimensional medical image model with simple motions of the input 
device. There Is a need for user-friendlier real time navigating systems in 
5 three-dimensional medical image models. 

Brief description of the invention 

An object of the invention is to provide an improved method and a 
system for navigating in real time in a three-dimensional medical image model. 
According to an aspect of the invention, there is provided a method for navigat- 

IQ, Jng in. real time jn a three, dimension rnethod com- 

prising: displaying an orientation view of the medical image model on a display; 
adjusting a location related to the displayed orientation view of the medical im- 
age model based on a pointing device alignment; displaying an inside view 
related to the location into the medical image model; and adjusting a viewing 

15 direction to the inside view of the medical image model based on a detected 
orientation of the pointing device. 

According to another aspect of the invention, there is provided a 
system for navigating in real time in a three-dimensional medical image model, 
the system comprising a control unit for controlling the functions of the system, 

20 a pointing device connected to the control unit and a display connected to the 
control unit, the control unit being configured to: display an orientation view of 
the medical image model on the display; adjust a location related to the dis- 
played orientation view of the medical image model based on the pointing de- 
vice alignment; display an inside view related to the location into the medical 

25 image model; and adjust a viewing direction to the inside view of the medical 
image model based on a detected orientation of the pointing device. 

Preferred embodiments of the invention are described in the de- 
pendent claims. 

The method and system of the invention provide several advan- 
30 tages. The viewing of the three-dimensional medical image models becomes 
simple. A large amount of information can be viewed very efRclently. Navigat- 
ing the details of a three-dimensional medical image model is possible in a 
user-friendly manner and even when the patient is not present. 
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List of drawings ' 

In the following, the invention will be described in greater detail with 
refbrenoe to the embodiments and the accompanying drawings, in which 

Figure 1 shows an example of the stnjcture of a system for navigat- 
5 ing in a three-dimensional medical image model; 

Figure 2 shows another example of the structure of a system for 
navigating in a three-dimensional medical image model, 

Figure 3 shows an example of the method for navigating in a three- 
dimensional medical image model. 
^0 Figures 4A and 4B show an example of the implementation of the 

method of navigating in a Ihree^limensional medical image model, and- 

Figures 5 and 6 show other examples of the implementation of the 
navigation method. 

Description of embodiments 

15 With reference to Figures 1 and 2, let us next study the examples of 

a system in which the embodiments of the invention can be applied. Figures 1 
and 2 illustrate the structure of the system for navigating in a three-dimensional 
medical image model. However, the embodiments are not limited to the sys- 
tems described in these examples; on the contrary, a person skilled in the art 

20 Is able to apply the Inventive solution also to other systans. 

The system 100 for navigating in real time in a three-dimensional 
medical Image model shown In Figures 1 and 2 comprises a control unit 102, a 
display 104, a pointing device 106 and a memory 108. 

The control unit 102 is connected to the display 104, to the pointing 

25 device 106 and to the memory 108. The control unit 102 refers to blocks con- 
trolling the operation of the system 100, and is nowadays usually implemented 
as a processor and software, but also different hardware implementations are 
feasible, e.g. a circuit built of separate logics components or one or more cli- 
ent-specific Integrated circuits (Application-Specific Integrated Circuit, ASIC). A 

30 hybrid of these Implementations is also feasible. The control unit 102 accesses 
the memory 108 during executing operations of the system. 

Typically, the display 104 is a color display monitor. In one embodl- 
ment of the invention, the display 104 is implemented with a contact surface 
thus forming a touch screen. In the touch screen, the contact suriiace is on top 
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of the display 104. for example. The control untt 102 displays images on the 

display 104^^ pointing device 106 comprises means with which the user is 
able to use the system 100. There can additionally be other user Interface 
pai such as a Jyboard or a mouse in the system 100. In the embo^men 
shown in Figure 1. the pointing device 106 is for example a pen. a pystck^a 
stylus or a track ball, which provides Input signals to the control umt 102. The 
input signals are Infomiation about the orientatlcn of the poinfng device 106 
for example. Also. Infomiation about a position and orientation such as a ti 
angle and pressure of the pointing device 106 are provided to the control unit 
-4oLn an-embodiment shown in Figure 2.-tiie-pointing-device 106 com^^^^^^^^^^ 

tablet 106A and a pen 106B. The tablet 106A is. for example, a srap^'^ tab- 
let such graphics tablet may be. for example, an Intuos 2 graphics tablet in- 
cluding a tilt sensitive pen that is manufactured by Wacom Co. Ltd. In one em- 
bodiment, tiie tablet 106A may be a contact sensitive graphics tablet and the 
pen i06Blsawireless pen. for example. The tabletlOOA provides in rea t.rne 
pos-rtion infomiation of the pen 106B tip 103 on the tablet 106A surface to the 
Ltrol unit 102. Also, information about tiie pen 1068 tilt, onentation and 
^essure Is provided to the control unit 102 by means of tiie tablet 106A and 
the pen 106B^ ^^^^ ^^^^ ,^ 

tern, for example. The memory 108 of the system may ^"^Pn^^ 

blocks 110 - 116. in which different data is stored. The memory blocks 110 - 

116 comprise, for example, annotated data from eariier sessions onentaticn 

, liLl^ata 3D medical image data sets and representative wori<n.g resulte^ 
sul as optimal surgical trajectory data. It is possible that the medical image 
model is created of two-dimensional medical image slices in tiie system. The 
rdilnsiona. medical Image slices or three-dimensional -^-^^^^^ 
models are transferred to the system by means known per se. for example. 

, from another system, such as PACS (Picture Archiving 

tem) or a device. The three-dimensional medical image model is created of 
dimensional MRl medical image slices, for example. It is Possjb^e that a 
Tumber of MRl image slices of the target area of a patient are obtained a 
niven!ntervais. The MRl image slices are taken so that the entire viewed target 

,5 lea of tiie patient is covered. As a result, a stack of two-dimensional image 
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slices is taken together thus outlining the entire three-dimensional volume of 
the target area. 

The control unit 102 is configured to display an orientation view of 
the three-dimensional medical image model on the display 104. The orientation 
5 view is, for example, a surface view of the medical Image model. Then a loca- 
tion related to the displayed orientation view of the medical Image model is 
adjusted based on the pointing device 106 alignment. The pointing device 106 
is» for example, a pen 106B and a tablet 106A, the pointing device 106 align- 
ment thus meaning the pen 106B tip 103 position, the pen 106B orientation or 

10 the pen 106B tilt angle on the tablet 106A surface. The location related to the 
displayed orientation view of the medical image model is a viewpoint or a point 
from which the user wishes to start navigating the three-dimensional medical 
image model. With the pointing device 106 alignments, the viewpoint to the 
model may be rotated thus causing the orientation view rotating at the same 

15 time. The user may, for example, move the pen 106B on the tablet 106A sur- 
face thus causing the orientation view of the three-dimensional medical image 
model to rotate horizontally on the display 104. The tilting of the pen 106B in 
relation to the tablet 106A surface may, for example, cause the orientation 
view of the three-dimensional medical image model to rotate vertically on the 

20 display 104. The speed and amount of the notation depends on the pen 106B 
tilt angle, for example. 

After the location is adjusted, the control unit 102 is further config- 
ured to display an inside view from the location into the three-dimensional 
medical image model. The inside view of the medical image model comprises 

25 one or more medical image slices or other reconstructions as seen from the 
selected location. The one or more medical image reconstructions are ren- 
dered with respect to the orientation view of the three-dimensional medical im- 
age model. As a given number of the medical image reconstructions are dis- 
played as the Inside view, an effect of navigating through the three- 

30 dimensional medical image model is created. The viewing direction to the in- 
side view of the three-dimensional medical image model is adjusted based on 
the orientation of the pointing device 106. The tilt angle of the pointing device 
106 is between the pen 106B and the tablet 106A surface, for example. The 
orientation view of the medical image model stays static while the viewing di- 

35 rection to the inside view of the medical image model is adjusted with the 
pointing device 106. 
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In an embodiment, It Is feasible that the Inside view into the three- 
dimensional medical image model proceeds deeper into the medical image 
model depending on a pressure against the pointing device 106. The pointing 
device 106 being a pen 106B and a tablet surface 106A, then the medical im- 

6 age model may proceed deeper into the medical image model depending on 
the pressure between the pen 106B and the tablet surface 106A. At the begin- 
ning of the navigation Into the Inside view only the first few medical image 
slices near the surface of the three-dimensional medical image model are dis- 
played, for example. Then, when the pen 106B tip 103 is pressed against the 

10 tablet 106A surface or an additional adjustment device 105, such as a joysticl< 
or a thumbwheel of the pen 1068 is used, for example, the inside view 
changes such that the image slices deeper in the three-dimensional medical 
image model are displayed instead. The adjustment device 105 Integrated to 
the pen 1068 may be Independent of the pen 1068 orientation and move- 

15 ments. With the adjustment device 105 different parameters of the three- 
dimensional medical image model, such as depth, contrast, transparency 
and/or threshold of the navigated image slices may be adjusted independent of 
the orientation of the pen 106B. For example, turning the thumbwheel may be 
used to adjust the viewpoint to the inside view Into the desired depth and to 

20 remain in that depth regardless of the movements of the pen 1 068. 

The displayed medical Image reconstructions or slices of the Inside 
view are two-dimensional, for example. The orientations of the medical Image 
reconstructions displayed on the display 104 are related to the axis of the pen, 
for example. The orientations of the medical image reconstructions are se- 

25 lected with the pointing device 106 or by other user interface parts, for exam- 
ple. 

Figure 3 shows an example of the method for navigating in a three- 
dimensional medical image model. The dashed lines are illustrating an optional 
method step. The method starts in 300 wherein the navigation system Is ready 
30 for use. The desired three-dimensional medical image model is selected and in 
302. the orientation view of the three-dimensional medical image model is dis- 
played on the display. 

In 304. the control unit detects the pointing device alignment. The 
control unit detects the pointing device movement and / or orientation. When 
35 the pointing device Is a pen and a tablet, then the pen tip movement on the 
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tablet surface and the pen tilt angle with reference to the tablet surface is de- 
tected. 

In 306, the location related to the orientation view of the three- 
dimensional medical image model is adjusted. The adjustment is carried out 

5 with the pointing device. The three-dimensional medical Image model is ro- 
tated vertically, horizontally and / or laterally by means of the pointing device. 
When the pointing device is a pen and a tablet, then the tilting of the pen in 
relation to the tablet would cause the viewpoint to the three-dimensional medi- 
cal image model to rotate laterally or vertically, for example. The moving of the 

10 pen tip on the tablet surface would in tum cause the three-dimensional medical 

image model to rotate horizontallyv for example^ - 

The reference point selection, In 308, Is an alternative method in 
one embodiment of the invention. The reference point Is displayed on the dis- 
play on the orientation view of the three-dimensional medical Image model. As 

16 the pointing device Is moved or tilted, the reference point also changes position 
on the orientation view of the three-dimensional medical image model. The 
reference point may be displayed with a cursor or the like on the display. When 
the reference point is at a desired place, at a possible treatment area, for in- 
stance, the reference point is selected to act as a navigation point. The control 

20. unit based on the pointing device, for example, detects the selection of naviga- 
tion point 

As the desired orientation of the three-dimensional medical image 
model is adjusted then, in 310, the control unit detects a start of a navigation 
mode. The start of the navigation mode is detected based on an input, for ex- 

25 ample, from the pointing device. If the start of the navigation mode is not de- 
tected then 304 and 306 may be executed. When the start of the navigation 
mode is detected based on depressing a button of the pointing device, for in- 
stance, then 312 is entered. In 312, the Inside view of the three-dimensional 
medical image model is displayed on the display. The inside view of the three- 

30 dimensional medical image model comprises one or more medical image re- 
constructions, for example, and the inside view into the medical image model 
is displayed related to the location of a navigation point. The number of medi- 
cal image reconstructions displayed on the display can be predetermined In 
the settings of the navigation system, for example. It is also feasible that the 

35 number of the medical Image reconstructions displayed on the display is al- 
tered during the navigation itself. 
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In 314, the viewing direction to the inside view of the three- 
dimensional medical image model is adjusted based on a detected orientation 
of the pointing device. The orientation of the pointing device may be based on 
the detected tilt angle and direction between the pen and the tablet surface, for 

5 example. The tilting of the pointing device causes the orientation of the one or 
more displayed medical Image slices to change, for example. The tilt angle is 
determined based on the pointing device orientation, for example. The pointing 
device being a pen and a tablet, then the tilt angle would be between the pen 
and the tablet surface, for instance. It is also feasible that more medical image 

10 reconstructions deeper in the three-dimensional medical image model are dis- 
played when pressure against the pointing device is-detected... For. .instance, 
the user of the navigation system can press a pen tip against a tablet surface 
and thus cause other medical Image reconstructions In different depths than 
the previously displayed medical image slices to appear on the display. 

15 In an embodiment, it Is possible that the medical image model com- 

prises one or more medical image slices and the adjusting of the viewing direc- 
tion to the inside view of the medical image model comprises rendering of the 
medical image slices with respect to the location related to the displayed orien- 
tation view of the medical image model. The medical image slices may be 

20 generated from two-dimensional image data, for example. The rendered medi- 
cal Image slices are oriented in relation to the detected orientation of the point- 
ing device, for example. In an embodiment of the Invention, the rendered 
medical image slices are orthogonal planes, for example three planes, one of 
the planes being perpendicular with the axis oriented in relation to the detected 

25 orientation of the pointing device. 

In 316, if the control unit detects the end of the navigation, then 318 
is entered wherein the navigation ends. Othenwise, 310 and 312 may be re- 
peated, it is also possible, that while navigating the user wishes to adjust the 
location related to the displayed orientation view again. This is possible in a 

30 situation where the user, such as a neurosurgeon, discovers, during the navi- 
gation mode, that the location related to the displayed orientation view of the 
medical image model selected in 306. should be changed. If the medical image 
model illustrates a patient's brain having a tumour, for example, then the sur- 
geon may wish to search for an optimum approach to the tumour in order to 

35 plan a surgical operation, for example. Then, from 320 it is moved to 302, 304 
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and 306, in which the orientation view orientation is adjusted, and then back to 
312 and 314. in which the actual navigation is perfonned. 

The user may adjust the location related to the displayed orientation 
view several times during the navigation in order to find an optimum view for 

5 the navigation. For example, a brain of a patient comprises many important 
neurological centres that have to be avoided. Also, the intracranial space 
around and within the brain is full of fluid spaces that are filled with clear cere- 
brospinal fluid. It is possible to move within these spaces from point to point 
using many surgical techniques including microsurgery and endoscopy. The 

10 method, as described above, offers a simple way to intuitively plan these com- 
plex surgical operations. 

It is possible, that the user adjusts the displayed orientation view by 
giving signals with an input device and uses another input device, such as the 
pen pointing device to navigate. Thus, several different input devices may be 

15 used for controlling the navigation method. The user may adjust the displayed 
orientation view at any time, even at the same time during the navigation 
mode. 

Figures 4A and 4B show examples of the implementation of the 
method of navigating in a three-dimensional medical image model. The Figure 

20 4A shows an example of how to select the reference point 401 A and the navi- 
gation point 401 B on the orientation view 400 of the three-dimensional medical 
image model shown on the display. In Figure 48, it is illustrated how the point- 
ing device 106B may be moved along the surface of the tablet 106A. The ar- 
rows 402 and 403 are illustrating how the pointing device 106B may be tilted in 

26 relation to the tablet 106A. The pointing device 106B may be moved along the 
surfece of the tablet 106A in any desired directions, for example. 

As the pointing device 106B is aligned on the tablet surface, by 
moving or tilting, for example, the reference point 401 A on the orientation view 
of the three-dimensional image model moves accordingly. When the desired 

30 point is found, this point is locked as a navigation point 401 B. In Figure 4A, the 
dashed an-ow is illustrating the route that the place of the reference point 401 A 
has been travelling before the locking of the navigation point 401 B. Then, as 
the pointing device 106B is aligned, by tilting for example, an inside view 
based on the alignment of the pointing device 1 06B is shown on the display. 

35 Figure 5 illustrates examples of inside views shown on the display 

104. The pointing device 106B is shown in dashed lines because typically it is 
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not shown In the Inside view of the medical Image model. However, it is also 
possible that a symbol representing the pointing device is shown on the display 
104 as well In Figure 5. the selected navigation point 4G1B is shown on the 
orientation view 400 of the medical image model. A circle like pattern illustrates 
the medical image slice 600 that is shown on the display 104. A frame 500 
around the medical image slice 600 Is for dartfylng the orientation of the medi- 
cal image slice 600 with reference to the orientation view 400 of the medical 

image model. u „ 

Another inside view window 502 Is shown in Figure 5. wherein 
frames 504. 506, 508 and medical image slices 510. 512. 514 from different 
points of view are shown. The Inside view window 502 consists of orthogonal 
slices 510 512. 514 along and across the axis of the pointing device 106B. In 
the medical image slices 510. 512. 514 there is also shown the place of the 
navigation point 401B in each of the slices 510. 512. 514. The lines departing 
from the navigation point 4018 In frames 504 and 506 are thus continuations 
of the axis of the pointing device inside the 3D medical image model. The 
medical Image slices 510 and 512 in frames 504 and 506 are orthogonal slices 
along the axis of the pointing device. In this example, the medical image slice 
514 in frame 508 is a slice perpendicular to the axis of the pointing device 
106B and the depth of the medical image slice 514 may be adjusted, for ex- 
ample by using the thumbwheel as described above. In the exemplary em- 
bodiment of Figure 5. the medical Image slice 514 Is the same medical image 
slice 600 that is shown on the left side of the display 1 04. 

in Figure 6. there is illustrated how the movement of the pointing 
device may cause the orientation of the medical image slice to change on the 
display with respect to the orientation view 400 of the medical image model. At 
first the pointing device's position Is as illustrated by the dashed lines num- 
ber^ with 1061. The medical image slice 600 and the frame 500 merited with 
dashed lines are illustrating the orientation of the slice 600 and the frame 500 
when the pointing device's position is at 1061. The arrow 604 shows how the 
pointing device's position changes from 1061 to 1062. In this example, the 
pointing device is being tilted upwards. The movement of the pointing device 
causes the orientations of the medical image slice 600 and the frame 500 to 
change. The new orientations of the medical image slice 602 and the frame 
35 502 are shown with continuous lines. Thus, the moving of the pointing device 



25 



30 
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causes the medical image slice to change orientation in relation to the orienta- 
tion view 400 of the medical image model. 

In Figures 5 and 6. there was shown one medical image slice with 
reference to the orientation view of the medical image model as an example. 

5 However, it Is possible that more than one medical image slice or medical im- 
age reconstruction set is shown at the same time on the display while navigat- 
ing the 3D medical image model. The number of medical image slices or re- 
constructions may be predetemiined by the user of the system or changed dur- 
ing the actual navigation. In an embodiment of the invention, also such is pes- 

10 sible that some of the medical image slices shown on the display are shown 
using different filtering or display parameters than the other medical image 
slices, for example. 

In an embodiment, it is also feasible that any data related to the 
navigated three-dimensional medical image model is recorded to the memory 

15 of the system. Such data may comprise one or more images, audio, video, an- 
notation data or any combination thereof. Thus, the data may comprise medi- 
cal image slices or reconstruction sets at any desired viewpoints and also an- 
notation related to the images. The user, such as a surgeon, may wish to re- 
cord such data at any time while navigating the three-dimensional medical im- 

20 age model. It is possible to record the whole navigation session including an- 
notations on the important items made by a surgeon while navigating the 
medical image model. The recording may comprise a video, movement or dis- 
playing parameters or any other parameters needed to reconstruct the com- 
plete navigation session later. Thus, a user of the system can make compre- 

25 hensive records of the navigation sessions and use the records at any time 
later, especially during a later surgical planning session when the patient is 
present, for example. 

The method provides a possibility to neuroradiological conferences 
that can be used to plan the procedures, such as an approach to a tumour and 

30 the work within the tumour. The surgical plan can be made during the naviga- 
tion, as a natural part of it, with the neurosurgeon consulting the neuroradiolo- 
gist, for instance. The data that has been recorded during the navigation may 
be accessed later and be shown in many clinical settings, including neurocon- 
sultation, surgical planning and patient education. Also, different parties, such 

35 as experts of certain medical areas, may add any annotations to the recorded 
data later. The recorded data may be printed out or saved as part of patient 
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medical history files. This makes it possible to easily show to a patient or to an 
insurance company, for example, any relevant infomnatlon about the medical 
procedures concerning an individual patient. For example, any risks and com- 
plications that may be involved in individual procedures may be recoreied. The 

5 recorded data relating to the navigation method may also comprise data about 
the patient's approvals to take the certain risks and complications involved In 
certain procedures. The data and the navigation method may also be easily 
used for educational purposes. 

Even though the invention is described above with reference to an 

10 example according to the accompanying drawings, It is clear that the invention 
Is-not-Festricted -thereto but-it-ean be modified- in several ways-withirv the scope 
of the appended claims. 



